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Executive  Summary 

The  ability  to  monitor  and  measure  brain  activity  patterns  in  individuals  undergoing  cognitive 
tasking  within  dynamic  environments  has  become  an  emerging  priority  across  military  and 
neuroscientific  communities,  particularly  those  interested  in  real-time  cognitive  monitoring, 
rehabilitation,  and  training.  A  variety  of  mobile  electroencephalographic  recording  systems  have 
been  developed  to  meet  this  demand;  however,  little  to  no  published  literature  is  available  that 
documents  efforts  to  validate  use  of  these  systems  under  complex  and  dynamic  conditions. 

This  report  summarizes  efforts  by  the  Naval  Health  Research  Center  to  systematically  test  and 
evaluate  the  performance  of  three  mobile  electroencephalography  (EEG)  systems,  in  both  static 
and  dynamic  conditions,  as  subjects  performed  classic  visual  and  auditory  oddball  tasks  within 
the  Physical  and  Cognitive  Operational  Research  Environment.  The  mobile  EEG  systems 
included: 

1.  Wearable  Sensing  DSI-24 

2.  Advanced  Brain  Monitoring  B -Alert  X24 

3.  ANT  Neuro  eego™sports  64 

Data  are  presented  on  how  these  mobile  EEG  systems  compared  with  one  another,  as  well  as 
against  a  "gold  standard"  wired  EEG  system,  including  data  on  preparation  times,  subjective  pain 
and  comfort  ratings,  and  system  capability  to  acquire  high-quality  EEG  signals.  Information  is 
also  presented  on  the  technical  integration  process,  with  recommendations  for  how  users  might 
mitigate  challenges  associated  with  synchronizing  the  presentation  of  environmental  stimuli  with 
mobile  EEG  signal  measurements  with  millisecond  precision. 

The  results  of  this  work  indicated  that  all  three  mobile  EEG  systems  were  capable  of  acquiring 
high-quality  EEG  signals;  however,  significantly  fewer  experimental  trials  were  needed  when 
using  the  ANT  system  to  acquire  the  event-related  potential  signal  in  both  the  resting  and  active 
conditions.  The  ANT  Neuro  system  also  rated  high  on  comfort,  low  on  pain,  and  within 
acceptable  limits  for  preparation  time.  Based  on  these  findings,  it  is  recommended  that  among 
the  three  mobile  EEG  systems  evaluated,  future  users  should  pursue  the  ANT  Neuro 
eego™sports  64  mobile  EEG  system  as  the  preferred  choice  for  use  in  complex  and  dynamic 
environments. 
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Introduction 

Cognitive  competence,  defined  here  as  the  ability  to  effectively  perform  the  cognitively  complex 
tasks  associated  with  normal  daily  living,  is  fundamental  to  meeting  military  requirements.  To 
evaluate  the  cognitive  competence  of  our  warfighters,  and  to  advance  the  development  of 
cognition-based  training  and  rehabilitation  programs,  systematic  investigations  of  cognitive 
competence  are  essential. 

Of  the  many  methods  for  measuring  cognitive  competence,  electroencephalography  (EEG)  is 
among  the  most  promising.  EEG  signals  provide  objective,  real-time  measurements  of  brain 
electrical  activity  patterns  with  markedly  high  temporal  resolution.1 II. III. IV. V.'6  Analyses  of  these  signals 
have  demonstrated  that  EEG  systems  are  capable  of  detecting  a  number  of  neural 
correlates/signatures  (i.e.,  neuromarkers)  for  a  variety  of  operationally  relevant  cognitive 
functions,  including  fatigue,  workload,  and  learning.7'20 

In  the  past  decade,  the  great  potential  of  EEG  measurements  for  monitoring  and  assessing 
cognitive  competence  has  promoted  the  rapid  development  of  mobile  EEG  systems  capable  of 
recording  real-time  brain  activity  in  a  variety  of  settings.  These  systems  are  marketed  as 
inexpensive,  portable,  easily  operable,  and  interpretable  without  technical  expertise.  However, 
most  of  these  mobile  EEG  systems  have  not  been  critically  evaluated  in  complex  or  dynamic 
environments.21"27 

To  systematically  address  this  gap,  a  comprehensive  investigation  was  carried  out  at  the  Naval 
Health  Research  Center  (NHRC)  in  San  Diego,  California,  to: 

1.  Identify  the  most  advanced,  commercially  available  mobile  EEG  systems, 

2.  Test  and  evaluate  these  mobile  EEG  systems  in  a  highly  complex  and  dynamic 
environment,  and 

3.  Provide  scientific  recommendations  and  technical  support  for  the  transition  of  the 
validated  mobile  EEG  system(s)  to  other  settings  within  the  Department  of  Defense 
(DoD). 

This  technical  report  summarizes  the  findings  of  this  effort  in  five  sections: 

I.  Description  of  the  EEG  Systems 

II.  Description  of  the  Testing  Environment 

III.  Integration  of  the  Mobile  EEG  Systems  Into  the  Testing  Environment 

IV.  Validation  of  the  Mobile  EEG  Systems 

V.  Recommendations 
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I.  Description  of  the  EEG  Systems 

Based  on  an  extensive  literature  review,  site  visits  to  the  vendors,  and  general  technical  inquiries, 
three  mobile  EEG  systems  were  selected  for  testing  and  evaluation  (Figure  1): 

1.  DSI-24  (Wearable  Sensing,  San  Diego,  CA,  USA) 

2.  B-Alert  X24  (Advanced  Brain  Monitoring,  Inc.,  Carlsbad,  CA,  USA) 

3.  eego™sports  64  (ANT  Neuro,  Enschede,  The  Netherlands) 


Figure  1.  Top  row:  Wearable  Sensing  DSI-24;  middle  row:  Advanced  Brain  Monitoring 
B-Alert  X24;  bottom  row:  ANT  Neuro  eego™sports  64. 
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The  decision  criteria  for  selecting  these  mobile  EEG  systems  included,  but  were  not  limited  to: 

•  Ease  of  use 

•  Durability  of  the  hardware 

•  Resilience  to  electrical  noise  interference  (signal-to-noise  ratio) 

•  Stability  of  the  acquired  brain  signals 

•  Capability  of  accepting  external  event  timing  markers 

•  Efficiency  and  reliability  of  the  data  acquisition  and  analysis  software 

a.  Wearable  Sensing  DSI-24 

The  DSI-24  (“WS”)  is  a  wireless,  research  grade,  dry  active  electrode  EEG  headset.  The  EEG 
sensors  are  located  on  21  pods  featuring  spring-loaded  finger  prongs  for  proper  fitting  (Figure  1, 
top  row).  The  upper  pods  are  connected  to  adjustable  sway  bars  to  provide  a  firm  connection  to 
the  scalp  for  head  circumferences  ranging  from  54-62  cm.  Electrode  placements  follow  the 
international  10-20  system  of  electrode  placement.  The  preamplifier  mounted  on  top  of  the 
headset  samples  EEG  signals  at  300  Hz;  it  then  transmits  the  raw  EEG  data  via  Bluetooth®  to  a 
data  acquisition  center.  The  amplifier  is  battery  operated  and  can  run  continuously  if  the  batteries 
are  “hot  swapped”  (i.e.,  one  of  the  two  batteries  can  be  replaced  while  the  system  is  in  data 
acquisition  mode).  The  WS  system  also  features  a  common-mode  follower  circuit  built  into  the 
headset  to  eliminate  a  large  portion  of  movement  artifacts. 

b.  Advanced  Brain  Monitoring  B-Alert  X24 

The  B-Alert  X24  (“ABM”)  features  an  absorbent  foam  and  electrode  gel  system  with  20 
channels  arranged  on  thin,  flexible  plastic  strips  (Figure  1,  middle  row).  The  plastic  strips  are 
attached  to  a  neoprene  headband  and  fitted  using  Velcro®  straps  to  accommodate  head 
circumferences  ranging  from  52-61  cm.  Electrode  placements  follow  the  international  10-20 
system.  The  preamplifier  mounted  on  the  back  of  the  headset  samples  EEG  signals  at  256  Hz  and 
then  transmits  the  raw  EEG  data  via  Bluetooth®  to  a  data  acquisition  center.  In  addition  to  the  20 
EEG  channels,  four  optional  channels  are  available  for  measuring  electrocardiography, 
electromyography,  or  electrooculography.  The  batteries  will  last  6  hours  using  Bluetooth®  or  16 
hours  if  data  are  recorded  directly  onto  a  secure  digital  card.  A  three-axis  accelerometer  is  built 
into  the  headset  for  monitoring  head  movement  and  position. 

c.  ANT  Neuro  eego™sports  64 

The  eego™sports  64  (“ANT”)  is  a  wet  gel  system  with  64  electrodes  embedded  within  a  soft 
EEG  cap,  the  waveguard™  (Figure  1,  bottom  row).  Three  sizes  of  the  waveguard™  are  available 
to  fit  most  adult  head  circumferences.  Each  electrode  is  surrounded  with  a  circular  gel  cup  to 
establish  an  electrical  connection  to  the  scalp.  Electrodes  are  placed  according  to  the 
international  10-10  system.  The  amplifier  is  connected  to  a  Sony  tablet,  both  of  which  fit  in  a 
small  backpack  carried  by  the  subject  for  mobility.  EEG  data  are  streamed  from  the  amplifier 
directly  to  the  tablet  for  storage.  The  sampling  rate  is  adjustable  to  a  maximum  of  2048  Hz.  The 
battery  of  the  amplifier  is  rated  to  last  5  hours;  however,  the  tablet  battery  lasts  approximately 
2.5  hours,  which  limits  the  maximum  allowable  recording  time.  The  tablet  communicates  via 
Wi-Fi  with  a  data  acquisition  center  to  control  the  operation  of  the  amplifier  and  the  data 
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acquisition  software. 

d.  Brain  Products  BrainAmp  DC  64 

In  addition  to  the  three  mobile  EEG  systems,  a  wired  EEG  system  (Figure  2)  was  acquired  to 
serve  as  a  “gold  standard”  against  which  the  mobile  systems  were  validated.  The  wired  EEG 
system,  BrainAmp  DC  64  (“BP”;  Brain  Products  GmbH,  Gilching,  Germany),  has  been  widely 
used  in  research  and  clinical  applications  (see  the  manufacturer’s  website  for  a  list  of  references). 
Therefore,  it  was  deemed  appropriate  for  referencing  the  validation  of  EEG  signals  recorded 
from  the  mobile  EEG  systems.  The  BP  is  a  wet  gel  system  with  64  electrodes  laid  out  on  the 
EASY  CAP  according  to  the  international  10-10  system.  Various  sizes  of  the  EASY  CAP  are 
available  to  fit  different  adult  head  sizes.  LED  lights  embedded  in  each  electrode  display 
different  colors  to  indicate  the  electrode  impedance  based  on  preset  threshold  values.  The  LEDs 
also  serve  as  land  markers  for  three-dimensional  (3-D)  sensor  localization  using  the  CapTrak 
scanner.  The  electrodes  are  connected  via  wired  ribbons  to  a  control  box  and  then  to  the  battery 
powered  BP  amplifier.  Digitized  EEG  signals  are  sent  via  optic  fibers  to  the  data  acquisition 
computer.  The  sampling  rate  is  adjustable  to  a  maximum  of  5000  Hz. 


Figure  2.  Brain  Products  BrainAmp  DC  64. 


Table  1  provides  a  side-by-side  comparison  of  many  of  the  technical  features  of  the  mobile  EEG 
systems.  Since  these  technologies  undergo  periodic  updates  and  advancements,  potential  users 
are  encouraged  to  visit  the  manufacturers’  websites  for  the  most  current  technical  specifications. 
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Table  1.  Technical  Features  of  the  Mobile  EEG  Systems 


ws 

ABM 

ANT 

Sensor  Type 

Active  dry  sensors 

Gel  on  absorbent  foam 

Gel  on  sensor  cups 

EEG  Channels 

20  (international  10-20 
system) 

20  (international  10-20 
system) 

64  (international  10-10  system) 

Weight 

700  g  (headset  and 
amplifier) 

110  g  (headset  and  amplifier) 

1700  g  (200  EEG  cap,  500 
amplifier,  500  backpack,  and  500 
g  tablet) 

Sampling  Rate 

300  Hz 

256  Hz 

Adjustable  up  to  2048  Hz 

Battery  Life 

Unlimited  run-time  with 
two  hot-swappable 
batteries 

6  h  using  Bluetooth®;  16  h 
with  secure  digital  card 

5  h  for  the  amplifier;  2.5  h  for 
the  tablet 

Durability 

Measure 

Sensor  attachment  bars 
are  subject  to  damage  if 
not  handled  carefully 

Potential  for  creases  in  the 
plastic  electrode  strip  and 
buildup  of  adhesive  on  the 
electrodes;  the  neoprene 
connecting  the  headband  and 
electrode  strip  is  prone  to 
tearing 

Gradual  buildup  of  dried  gel 
within  the  electrode  wells  if  not 
cleaned  thoroughly  after  each 
use 

Mobility 

Transmits  up  to  10  m  via 
Bluetooth®;  requires  a 
cable  for  stimulus  event 
time  stamps 

Transmits  up  to  10  m  via 
Bluetooth®,  or  records 
directly  onto  secure  digital 
card;  requires  a  cable  for 
stimulus  event  time  stamps 

Records  directly  onto  tablet 
carried  by  subject;  tablet  is 
controlled  via  Wi-Fi;  requires  a 
cable  for  stimulus  event  time 
stamps 

Impedance 

Stability 

Stable  throughout  the 
recording  session 

Stable;  progressively 
decreases  as  recording 
sessions  continue 

Stable;  progressively  decreases 
as  recording  sessions  continue 

Headset  Size 
(Circumference) 

52-62  cm 

51-56  and  56-61  cm 

47-51,51-56,  and  56-61  cm 

Scalp  Locations  of 
Electrodes 

Fpl,  Fp2,  Fz,  F3,  F4,  F7, 
F8,  Cz,  C3,  C4,  Pz,  P3, 

P4,  T3,  T4,  T5,T6,  01, 
02,  Ml,  M2 

Fpl,  Fp2,  Fz,  F3,F4,  F7,  F8, 
Cz,  C3,  C4,  Pz,  P3,  P4,  T3, 

T4,  T5,T6,  POz,  01,02 

Fpz,  Fpl,  Fp2,  AFz,  AF3,  AF4, 
AF7,  AF8,  Fz,  FI,  F2,  F3,  F4, 

F5,  F6,  F7,  F8,  FCz,  FC1,  FC2, 
FC3,  FC4,  FC5,  FC6,  FT7,  FT8, 
Cz,  Cl,  C2,  C3,  C4,  C5,  C6,  T7, 
T8,  CPz,  CPI,  CP2,  CP3,  CP4, 
CP5,  CP6,  TP7,  TP8,  Pz,  PI,  P2, 
P3,  P4,  P5,  P6,  P7,  P8,  POz, 

P03,  P04,  P05,  P06,  P07, 

P08,  Oz,  01,  02,  Ml,  M2 
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II.  Description  of  the  Testing  Environment 

The  Computer  Assisted  Rehabilitation  Environment  (CAREN;  Motekforce  Link,  Amsterdam, 
The  Netherlands;  Figure  3)  is  a  fully  immersive  virtual  reality  system.  The  DoD  presently 
operates  four  C ARENs  in  the  following  centers: 

1.  NHRC 

2.  National  Intrepid  Center  of  Excellence,  Bethesda,  Maryland 

3.  Walter  Reed  National  Military  Medical  Center,  Bethesda,  Maryland 

4.  Center  for  the  Intrepid,  Fort  Sam  Houston,  Texas 


Figure  3.  Computer  Assisted  Rehabilitation  Environment  (CAREN)  Extended  model. 


The  CAREN  Extended  model  at  NHRC  features  numerous  measurement  and  presentation 
upgrades  from  the  standard  model  to  produce  clinical  and  operationally  relevant  programs.  This 
system  consists  of  a  full  motion,  9-foot-diameter  platform  programmable  to  move  in  6  degrees  of 
freedom,  independently  or  simultaneously.  At  the  center  of  the  platform  is  a  high  performance 
(i.e.,  large  acceleration  and  reverse  mode),  dual-belt  treadmill  with  integrated  force  plates 
underneath  to  measure  ground  reaction  forces.  The  platform  is  surrounded  by  a  180-degree  wide 
and  9-foot  tall  curved  screen,  equipped  with  a  14-camera  optical  motion  capture  system  for 
tracking  movement  of  the  subject.  This  CAREN  also  features  three  3-D  video  projectors, 
surround  sound  speakers,  and  a  programmable  scent  delivery  device,  which  together  provide 
fully  immersive  experiences  with  multimodality  sensations  (visual,  auditory,  vestibular,  and 
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olfactory)  that  can  be  controlled  by  the  subject  or  operator.  Based  on  NHRC  CAREN  updates 
and  programmatic  capabilities,  it  has  been  renamed  the  Physical  and  Cognitive  Operational 
Research  Environment,  or  PhyCORE  (Figure  4).  This  system  provides  an  ideal  platform  for 
integrating,  testing,  and  evaluating  the  mobile  EEG  systems  within  highly  controlled,  repeatable, 
and  dynamic  task  conditions.28’29 


Figure  4.  The  Physical  and  Cognitive  Operational  Research  Environment  (PhyCORE;  top  left), 
with  custom  driving  simulator  (top  right),  military-relevant  scenarios  (bottom  left),  and  multiple 
biosensors  providing  real-time  monitoring  and  measurements  (bottom  right). 


III.  Integration  of  the  Mobile  EEG  Systems  Into  the  Testing  Environment 

To  effectively  evaluate  the  performance  of  the  mobile  EEG  systems,  the  first  task  was  to 
establish  digital  communication  pathways  through  which  the  recorded  EEG  signals  could  be 
accurately  time  stamped  with  events  occurring  in  the  PhyCORE.  Technical  challenges  associated 
with  this  integration  process  included: 

1.  Establishing  a  mechanism  by  which  the  PhyCORE  Control  Center  could  transmit 
event  signals, 

2.  Customizing  the  process  by  which  each  mobile  EEG  system  would  receive  event  time 
stamps,  and 

3.  Calibrating  the  timing  accuracy  and  jitters  embedded  in  the  established  digital  signal 
pathways. 
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Approaches  for  overcoming  these  technical  challenges  were  presented  at  the  2015 
Interservice/Industry  Training,  Simulation  and  Education  Conference  (I/ITSEC)  in  Orlando, 
Florida,  and  published  in  the  conference  proceedings  (Appendix  A).  Main  solutions  are 
summarized  below. 

The  greatest  technical  challenge  in  integrating  the  mobile  EEG  systems  into  the  PhyCORE  was 
determining  how  to  time  stamp  the  EEG  signals  with  PhyCORE  events.  Practical  solutions 
depended  heavily  on  how  the  event  signals  were  transmitted  from  the  event  generators  and  how 
the  signals  were  received  by  the  EEG  systems.  Complications  arose  from  the  inherent  design 
differences  among  the  mobile  EEG  systems,  since  each  system  had  unique  methods  for 
achieving  mobility  and  synchronizing  external  events  with  the  EEG  signals. 

As  depicted  in  Figure  5,  for  the  ABM  system,  EEG  signals  were  acquired  in  the  headstage  unit 
and  transmitted  via  Bluetooth®  to  an  External  Sync  Unit  for  time  stamping  the  EEG  signals.  The 
time  stamped  EEG  signals  were  then  streamed  into  the  data  acquisition  center  through  a  USB 
cable.  However,  the  use  of  Bluetooth®  transmission  prior  to  time  stamping  created  an 
unavoidable  time  delay  (Tbiue)  of  approximately  35  ms  and  averaged  jittering  of  20  ms,  as 
reported  by  the  manufacturer.  In  contrast,  the  external  event  signals  for  both  the  WS  and  ANT 
systems  were  sent  directly  to  an  EEG  amplifier  through  a  cable  connected  to  the  event  generator 
(PhyCORE  Control  Center).  Thus,  for  these  systems,  EEG  signals  were  time  stamped  prior  to 
their  wireless  transmission  to  the  data  acquisition  computer.  This  method  effectively  eliminated 
the  Tbiue  found  with  the  ABM  system. 


Amplifier 


ABM  WS 


EEG  signals  without 
event  time  stamps 

EEG  signals  with 
event  time  stamps 


blue 


nnji 


External  sync  j 
unit 


ANT 


PhyCORE 


D-Flow  commands 


JUUL 


Data  acquisition  center 


PhyCORE  Control  Center 


Figure  5.  Technical  approach  diagram  for  integrating  the  mobile  EEG  systems  into  the  PhyCORE. 
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From  the  event  generator,  events  such  as  platform  movement,  treadmill  speed,  and  stimulus 
presentation  were  all  programmed  and  produced  with  a  software  program  called  D-Flow 
(Motekforce  Link,  Amsterdam,  The  Netherlands).  By  design,  D-Flow  does  not  communicate 
with  the  computer’s  parallel  ports.  Therefore,  to  send  PhyCORE  event  time  stamps  to  the  EEG 
systems,  a  1018  Phidget  I/O  Board  (Phidgets  Inc.,  Calgary,  Alberta,  Canada)  was  used  to  receive 
event  time  stamps  via  a  USB  port.  This  particular  board  can  input  digital  or  analog  signals  and 
provide  a  digital  output  through  a  USB  cable.  The  Phidget  I/O  Board  sent  the  event  time  stamps 
to  a  PCI-based  digital  I/O  board  residing  in  the  computer  of  the  data  acquisition  center.  Only 
then  could  the  event  time  stamps  be  relayed  to  the  EEG  systems  through  the  parallel  ports. 
Inevitably,  this  event  timing  transmission  method  created  a  systematic  delay  (Tsys)  in  the  time 
stamping  of  EEG  signals  (Tsys  =  ti  +  t2).  This  Tsys  was  independent  of  the  design  of  the  EEG 
systems  and  was,  therefore,  consistent  for  all  three  systems.  Using  this  approach,  the  total  delay 
in  event  timing  for  the  WS  and  ANT  systems  was  Tsys.  For  the  ABM  system,  the  total  delay  was 
Tsys  +  Tbiue  (equivalent  to  ti  + 12  +  Tbiue)- 

The  existence  of  time  delays  was  inevitable  due  to  technical  restraints;  nevertheless,  the  value  of 
Tsys  could  be  measured  accurately.  As  shown  in  Figure  6,  Tsys  was  calibrated  using  a  trigger  box 
manufactured  by  Wearable  Sensing.  The  actual  timing  (Tact)  of  the  PhyCORE  screen’s  visual 


Lisht sensor 


Microphone  |  D-Flow  commands 


PhyCORE  Control  Center 


Figure  6.  Calibration  of  event  time  delays  and  jitters. 
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display  or  the  surrounding  speakers’  sound  emission  was  detected  by  a  light  sensor  or 
microphone  connected  to  the  trigger  box.  The  trigger  box  then  sent  the  visual  or  audio  event 
signal  to  the  headstage  (Figure  6,  top).  Simultaneously,  the  event  marker  signals  sent  to  the  EEG 
system  from  the  data  acquisition  center  were  also  sent  to  the  headstage  and  recorded  as  Tevt-  The 
values  of  Tsys  were  then  measured  off-line  by  calculating  the  difference  between  Tevt  and  Tact 
(Figure  6,  bottom).  The  mean  values  of  Tsys  were  21.9  ms  and  30.7  ms  for  the  visual  and 
auditory  stimulation,  respectively.  These  Tsys  values  were  accounted  for  in  EEG  data  analyses 
for  both  the  WS  and  ANT  systems.  For  the  ABM  system,  an  additional  35  ms  (Tbiue)  were  used 
to  compensate  for  the  delay.  Further,  for  each  system,  there  was  a  fixed  time  delay  associated 
with  the  EEG  signal  digitization  in  the  amplifier  that  needed  to  be  taken  into  account  during  data 
analyses.  The  duration  of  this  delay  varied  by  system  and  was  made  available  by  each  of  the 
system  manufacturers. 

In  summary,  a  technical  framework  for  integrating  the  mobile  EEG  systems  into  the  PhyCORE 
with  millisecond  time  synchronization  was  established  through  execution  of  this  project.  The 
three  mobile  EEG  systems  tested  in  this  work  represent  the  full  technical  scope  of  how  EEG 
signals  are  acquired,  wirelessly  transmitted,  and  time  stamped  in  the  industry.  This  approach  for 
integrating  mobile  EEG  systems,  although  developed  specifically  for  the  PhyCORE,  can  be 
generalized  and  adapted  by  other  types  of  immersive  virtual  reality  enviornments,  CARENs,  and 
clinical  settings. 

IV.  Validation  of  the  Mobile  EEG  Systems 

The  ultimate  goal  of  implementing  mobile  EEG  systems  into  the  PhyCORE  was  to  determine 
whether,  and  to  what  degree,  these  systems  could  detect  behaviorally  relevant  EEG  signals  under 
highly  complex  and  dynamic  conditions.  To  this  end,  a  research  proctocol  titled  “Validating 
Mobile  EEG  Systems  for  Integration  Into  the  PhyCORE  and  Application  in  Clinical  Settings,” 
was  developed  and  approved  by  the  institutional  review  board  at  NHRC  (Protocol 
NHRC. 2014.0017).  Preliminary  analyses  from  this  study  were  published  in  the  2015  I/ITSEC 
conference  proceedings  as  a  paper  (Appendix  A)  and  presented  as  three  conference  posters 
(Appendixes  B-D).  The  results  of  these  authored  works  and  additional  findings  are  detailed 
below. 

a.  Method  of  Assessment 

Twelve  healthy  volunteers  participated  in  the  study  (7  women,  5  men,  age  range:  21-40  years). 
Subjects  reported  individually  to  NHRC’s  PhyCORE  laboratory.  They  were  informed  that  with 
their  consent  they  would  be  fitted  with  three  different  mobile  EEG  headsets  and  undergo  similar 
testing  trials  on  the  PhyCORE,  while  wearing  each  system.  After  donning  each  headset,  an 
impedance  check  was  performed  in  accordance  with  each  vendor’s  impedance  test  utility  tool  to 
ensure  all  channel  connections  were  satisfactory.  Following  testing  of  each  system,  subjects 
completed  a  brief  survey  evaluating  their  levels  of  pain  and  comfort  using  rating  scales  and 
open-ended  questioning  for  each  factor.  Short  breaks  (10-20  min)  were  taken  between  testing 
conditions,  in  addition  to  a  1-hour  lunch  break.  Time  to  complete  the  entire  experiment  was 
approximately  8  hours. 

Testing  trials  consisted  of  classical  auditory  and  visual  oddball  experiments  performed  under  two 
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conditions:  (1.)  Resting :  subjects  sat  still  on  a  chair  located  on  the  stationary  PhyCORE  platform 
with  a  visual  display  of  a  stationary  nature  trail  scene;  and  (2.)  Active:  subjects  walked  at  a  self- 
selected  pace  on  the  PhyCORE  treadmill  with  a  visual  display  of  a  dynamic  nature  trail  scene 
that  flowed  in  sync  with  their  walking  speed.  Due  to  extensive  wiring,  trials  with  the  BP  system 
were  only  completed  in  the  resting  condition.  Preceding  each  testing  trial,  a  baseline  assessment 
of  each  subject’s  EEG  activity  was  acquired  while  the  subject  sat  with  eyes  opened  and  then 
closed  for  2  min  each. 

For  the  auditory  oddball  paradigm,  a  series  of  bird  chirps  was  presented  through  the  surround 
sound  speakers  as  the  frequent  stimulus,  and  toad  croaks  as  the  rare  stimulus.  For  the  visual 
oddball  paradigm,  a  bird  image  was  displayed  at  the  center  of  the  screen  as  the  frequent  stimulus, 
with  a  display  of  a  toad  image  as  the  rare  stimulus.  Presentation  of  frequent  to  rare  stimuli 
occurred  at  a  4:1  ratio.  The  stimulus  duration  was  0.5  s,  and  the  stimulus  interval  varied  pseudo- 
randomly  between  1. 5-2.0  s.  The  rare  stimuli  were  presented  pseudo-randomly  following 
presentation  of  2-10  frequent  stimuli.  A  total  of  250  (200  frequent,  50  rare)  and  500  (400 
frequent,  100  rare)  stimuli  were  presented  in  each  session  for  the  sitting  and  walking  conditions, 
respectively,  which  corresponded  to  about  10  min  for  each  sitting  trial  and  20  min  for  each 
walking  trial.  For  all  trials,  subjects  were  instructed  to  silently  count  the  total  number  of  frequent 
stimuli  and  report  this  total  at  the  end  of  each  session. 

b.  Operational  Feasibility  Evaluation 

Preparation  and  Cleanup  Time 

The  preparation  time  for  each  mobile  EEG  system  was  defined  as  time  elapsed  from  initiation  of 
the  fitting  process  to  completion  of  the  impedance  check.  The  average  time  duration  to  complete 
each  of  the  preparation  stages  for  each  system  is  summarized  in  Figure  7.  Overall,  the  dry 


Cleanup  time 
Impedance  check  time 
■  Setup  time 
4  Total  time 

Average  time  per  channel 


Mobile  EEG  Headset 

Figure  7.  Mean  preparation  and  cleanup  times  for  the  mobile  EEG  systems. 
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electrode  WS  system  required  less  total  preparation  time  than  the  other  two  wet  electrode 
systems.  Despite  large  differences  in  the  number  of  channels,  preparation  time  for  the  ANT 
system  (setup  +  impedance  check  =  25.4  min)  was  slightly  shorter  than  that  for  the  ABM  system 
(average  29.8  min).  However,  the  cleanup  time  for  the  ANT  system  (average  25  min)  was  much 
longer  than  the  other  two  systems  (ABM,  10  min;  WS,  3  min),  most  of  which  was  spent  on 
brushing  the  gel  out  of  each  of  the  64  electrode  cups.  Nonetheless,  among  the  three  mobile  EEG 
systems,  the  averaged  total  time  (preparation  and  cleanup)  spent  on  each  channel  was  lowest  for 
the  ANT  system.  Thus,  the  ANT  system  was  the  most  time-efficient  system  with  respect  to 
preparation  and  cleanup. 

Pain  and  Comfort  Ratings 

After  testing  each  of  the  three  mobile  EEG  systems,  subjects  completed  rating  scales  to  indicate 
their  levels  of  pain,  ranging  from  0  ( no  pain )  to  10  ( extreme  pain),  and  comfort,  ranging  from  0 
(very  uncomfortable)  to  5  (i very  comfortable),  experienced  with  each  system.  Means  and  standard 
deviations  for  these  assessments  are  presented  in  Table  2  and  Figures  8  and  9. 

Results  of  separate  one-way  analyses  of  variance  yielded  statistically  significant  differences  in 
comfort  (F( 2,  33)  =  10.07,  p  <  0.001)  and  pain  (F(2,  33)  =  13.79,  p  <  0.001)  ratings  among  the 
headsets.  Games-Howell  post  hoc  testing  indicated  the  WS  system  rated  significantly  higher  on 
the  pain  scale  than  the  ABM  and  ANT  systems  (both  p  <  0.05);  however,  the  mean  pain  rating 
for  the  WS  system  (2.17  on  a  10-point  scale)  was  still  fairly  low.  This  suggests  that  although  the 
WS  system  rated  higher  on  the  pain  scale,  it  was  not  necessarily  rated  as  painful.  The  mean  pain 
score  comparison  between  the  ABM  and  ANT  systems  was  not  statistically  significant  (in  fact, 
they  were  identical).  For  the  comfort  scale,  the  Games-Howell  post  hoc  test  indicated  the  WS 
system  was  rated  significantly  less  comfortable  than  the  ABM  and  ANT  systems  (both  p  <  0.05); 
the  latter  of  which,  again,  did  not  significantly  differ. 

Effect  size  analyses  (Cohen’s  d)  were  conducted  on  all  statistically  significant  findings  to 
determine  the  degree  to  which  pain  and  comfort  ratings  for  each  system  differed  in  terms  of 
practical  significance  (0.2,  small  effect;  0.5,  medium  effect;  0.8,  large  effect).  Based  on  these 
analyses,  the  WS  system  was  rated  as  considerably  more  painful  than  the  ABM  (d  =  1.33)  and 
the  ANT  (d  =  1.43)  systems  and  considerably  less  comfortable  (ABM,  d=  1.77;  ANT,  d  =  1.37). 
These  large  effect  size  estimates  support  the  notion  that  subjects  expressively  rated  the  WS 
system  as  more  painful  and  less  comfortable  than  the  ABM  and  ANT  systems  based  on  their  user 
experience. 


Table  2.  Pain  and  Comfort  Ratings 


Mobile 

EEG  Pain  Rating 

Comfort  Rating 

System 

M  (SD) 

M  (SD) 

WS 

2.17(1.85) 

3.17  (1.27) 

ABM 

0.25  (0.87) 

4.83  (0.39) 

ANT 

0.25  (0.45) 

4.50  (0.52) 
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Figure  8.  Perceived  pain  ratings. 
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Figure  9.  Perceived  comfort  ratings. 
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c.  EEG  Signal  Quality  Assessments 

Consistent  with  analyses  performed  in  the  EEG  literature,30’31  the  grand  average  event-related 
potential  (ERP)  was  computed  for  each  mobile  EEG  system.  The  ERP  waveforms  used  in  these 
analyses  were  obtained  by  averaging  multiple  EEG  epochs  temporally  aligned  to  auditory  or 


ABM  ANT  WS 


^“Frequent  Visual  ERP 


Figure  10.  Across  mobile  EEG  systems,  comparisons  of  the  grand  average  ERPs  acquired 
under  both  sitting  and  walking  conditions  from  the  auditory  (top  panel,  sensor  F3)  and  visual 
(bottom  panel,  sensor  P4)  oddball  experiments  from  all  12  participants. 
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visual  stimuli  presented  in  the  PhyCORE.  Through  this  process,  random  and  event-uncorrelated 
electrical  noises  were  averaged,  and  the  underlying  phase-locked  ERP  waveforms  were  revealed. 
As  depicted  in  Figure  10,  all  three  mobile  EEG  systems  yielded  adequate  quality  and 
behaviorally  relevant  ERP  waveforms. 

In  theory,  any  electric  signal  sensor  placed  on  the  scalp  can  be  used  to  measure  ERP,  given 
sufficient  repetitions  of  stimulus  presentation  or  motor  actions.  In  practice,  however,  there  is 
often  neither  time  nor  resources  to  allow  for  the  hundreds  or  even  thousands  of  events  that  may 
be  necessary  to  acquire  a  compelling  ERP  measurement;  therefore,  only  those  EEG  systems  that 
require  the  fewest  stimulus  presentations  or  motor  actions  to  generate  ERP  have  the  most 
pragmatic  value.  Therefore,  we  performed  usability  analyses  on  the  three  mobile  EEG  systems  to 
compare  the  number  of  stimulus  presentations  required  for  each  system  to  measure  ERP.  To 
assess  usability  of  the  three  systems,  judged  by  the  EEG  measurement  reliability  and  ERP 
quality  assurance,  a  two-step  bootstrap  method  was  developed  to  estimate:  (1)  the  minimal 
number  of  trials  required  for  detecting  the  presence  of  ERP,  and  (2)  the  optimal  number  of  trials 
needed  to  obtain  high-quality  ERP. 

To  quantify  the  overall  strength  of  the  ERPs  in  the  auditory  and  visual  oddball  paradigms,  an 
ERP  index  (idx)  was  calculated  from  the  formula: 


PoStRMS  -  Pl'CRMS 

idx  = - 

PoStRMS  +  Pl'CRMS 


where,  PostRMs  was  the  root  mean  square  (RMS)  of  the  ERP  amplitude  within  the  800  ms  time 
window  after  onset  of  the  sensory  stimulus,  and  PreRMs  was  the  RMS  within  the  400  ms  time 
window  before  onset  of  the  sensory  stimulus.  A  value  of  idx  <  0  indicated  the  absence  of  ERP, 
idx  =  0  suggested  no  more  ERP  signal  than  noise,  and  idx  >  0  signified  the  presence  of  ERP.  The 
extreme  values  of  idx  ranged  from  -1  to  +1  corresponding  to  PostRMs  =  0  and  PreRMs  =  0,  two 
unachievable  conditions,  respectively. 

An  example  of  ERP  obtained  from  the  active  auditory  oddball  paradigm  recorded  with  the  WS 
system  is  shown  in  Figures  11A  and  12A.  EEG  epochs  aligned  with  the  onset  of  sound  stimuli, 
including  both  frequent  and  rare  stimuli,  were  averaged  to  reveal  the  underlying  ERP  waveform. 
The  ERP  index  value  (idx  =  0.469)  was  closely  associated  with  the  high-quality  ERP  waveform 
shown  in  the  bottom  panel  of  Figure  1 1 A  and  Figure  12 A. 

A  single-point  measurement  of  idx,  however,  does  not  provide  any  information  about  the 
variability  of  EEG  signals  recorded  with  the  same  or  different  EEG  systems;  therefore,  it  could 
not  be  used  to  assess  the  mobile  EEG  systems’  usability  (i.e.,  EEG  measurement  reliability  and 
ERP  quality  assurance).  A  two-step  method  based  on  bootstrap  resampling  was  developed  to 
achieve  this  objective  (Figure  ll).32”44 

Step  One:  Bootstrap  resampling  of  randomly  selected  samples  (S)  out  of  the  number  of  EEG 
epochs  (AO  (Figures  11B  and  1 1C)  yielded  an  idx  probability  distribution  of  ERP  waveforms  for 
a  single  EEG  channel  (Figure  11D).  The  95%  confidence  interval  ( idx-CI)  was  computed  based 
on  this  probability  distribution.  The  lower  bound  of  this  idx-CI,  the  LBs,  provided  an  assessment 
of  EEG  measurement  reliability  for  yielding  an  ERP  signal  in  a  single  EEG  channel: 
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•  LBS  <  0,  unreliable  EEG  measures  and  ERP  absence 

•  LBs  >  0,  reliable  EEG  measures  and  ERP  presence 

The  value  LBs  =  0  can  thus  serve  as  the  threshold  for  determining  the  minimal  number  of  trials 
(MNT)  required  for  detecting  the  presence  of  ERP. 

Step  Two:  The  bootstrap  resampling  process  outlined  in  step  1  was  iterated  multiple  times  with 
different  resampling  sizes.  The  total  number  of  iterations  (i.e.,  number  of  bootstraps)  was 
determined  by  the  step  function:  [4:  2:  (A-l)]  (minimal  S  =  4,  maximal  S  =  N-l,  step  size  =  2, 
total  iterations  =  (iV-5)/2).  A  spline  regression  of  the  resulting  set  of  LBs  values  as  a  function  of 
resampling  size  S  was  performed  to  estimate: 

1.  The  MNT  required  to  detect  the  presence  of  ERP:  bootstrap  resampling  size  S 
corresponding  to  LBs  =  0  (Figure  11E),  and 

2.  The  optimal  number  of  trials  (ONT)  needed  to  obtain  high-quality  ERP:  bootstrap 
resampling  size  S  corresponding  to  LBs  =  0.170  (Figure  11E). 

The  value  LBS  =  0.170  was  calculated  based  on  the  estimated  minimal  signal-to-noise  ratio 
(PostRMs/PreRMs  =  3  dB)  of  the  ERP  waveform  to  be  considered  a  high-quality  signal  as  reported 
in  the  literature.45"49 

Figure  12  depicts  the  relationship  between  ERP  quality  and  LBs  values.  With  LBs  <  0  (Figure 
12B),  the  bootstrap  ERP  waveforms  (averaged  across  4  EEG  epochs)  did  not  assemble  at  all  to 
the  acquired  ERP  waveform  (averaged  across  all  322  EEG  epochs,  Figure  12A).  When  LBs  =  0 
(Figure  12C),  the  major  features  of  the  acquired  ERP  waveform  appeared  clearly  on  all  three 
examples  of  the  bootstrap  ERP  waveforms.  With  LBs  =  0.170  (Figure  12E)  and  halfway 
between  0  and  0.170  (LBS  =  0.080,  Figure  12D),  detailed  patterns  of  major  components  of  the 
acquired  ERP  waveform  became  clearly  visible.  Thus,  in  order  to  assess  usability  of  the  mobile 
EEG  systems,  the  bootstrap  sample  sizes  corresponding  to  the  values  LBS  =  0  and  LBS  =0.170 
were  used  as  objective  measures  for  determining  the  MNT  required  for  detecting  ERP  and  the 
ONT  needed  to  acquire  high-quality  ERP  waveforms,  respectively. 

The  MNT  required  for  detection  of  ERP  for  all  20  common  channels  across  the  12  subjects  in 
both  the  auditory  and  visual  oddball  paradigms  were  compared  separately  under  resting  and 
active  conditions  (Figure  13 A).  In  the  resting  condition,  the  MNT  for  the  WS  and  ABM  systems 
were  comparable  (WS:  117  +  16  SE,  n  =  123;  ABM:  116  +  12  SE,  n  =  141;  Wilcoxon  rank  sum 
test:  p  =  0.6661),  and  both  were  significantly  higher  than  the  MNT  for  the  ANT  (71  +  12  SE,  n  = 
136;  p  <  0.0001  for  both  WS  vs.  ANT  and  ABM  vs.  ANT)  and  BP  (62  +  8  SE,  n  =  60;  p  < 
0.0001  for  both  WS  vs.  BP  and  ABM  vs.  BP)  systems.  In  contrast,  the  MNT  for  the  ANT  system 
was  slightly  higher,  but  not  statistically  different,  than  the  MNT  for  the  BP  system  (p  =  0.4233). 
Thus,  in  the  resting  condition,  the  ANT  system  required  approximately  the  same  number  of  trials 
for  detecting  ERP  signals  as  the  gold  standard  wired  EEG  system  (i.e.,  BP),  while  the  WS  and 
ABM  systems  required  significantly  more  trials  for  detection  to  the  same  degree.  Similarly,  in 
the  active  condition,  the  MNT  for  the  ANT  system  was  significantly  lower  than  the  MNT  for 
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both  the  WS  and  ABM  systems  (WS:  199  ±  12  SE,  n  =  161;  ABM:  175  +  11  SE,  n  =  148;  ANT: 
147  ±  14  SE,  n  =  132;  p  <  0.0001  for  both  WS  vs.  ANT  and  ABM  vs.  ANT). 

Since  the  MNT  for  detecting  ERP  presence  is  directly  related  to  the  EEG  measurement  reliability 
and  signal  quality,  the  ONT  for  obtaining  high-quality  ERP  would  be  expected  to  be  lower  in  a 
system  with  lower  MNT  for  detecting  ERP  presence.  As  depicted  in  Figure  13B,  the  ONT  for  the 
ANT  system  was  indeed  significantly  lower  in  both  the  resting  and  active  experimental 
conditions  than  the  ONT  for  both  the  WS  and  ABM  systems  (resting  WS:  192  +  22  SE,  n  =  53; 
ABM:  171  +  19  SE,  n  =  51;  ANT:  128  +  17  SE,  n  =  96;  p  <  0.0001  for  both  WS  vs.  ANT  and 
ABM  vs.  ANT;  active  WS:  275  +  20  SE,  n  =  84;  ABM:  279  +  15  SE,  n  =  80;  ANT:  176  +  15  SE, 
n  =  95;  p  <  0.0001  for  both  WS  vs.  ANT  and  ABM  vs.  ANT).  In  the  resting  condition,  the  ONT 
for  the  ANT  system  was  slightly  higher  than  that  for  the  BP  system  (ANT:  128  +  17  SE,  n  =  96; 
BP:  115  +  12  SE,  n  =  36),  but  this  difference  was  not  statistically  significant  (p  =  0.2634). 

Using  two-step  bootstrap  resampling  of  real  data  recorded  from  the  three  mobile  EEG  systems 
and  a  wired  gold  standard  EEG  system,  a  fully  objective  and  statistically  rigorous  method  was 
established  for  assessing  EEG  measurement  reliability  and  ERP  quality  assurance  with  two 
simple  and  straightforward  parameters:  the  MNT  for  detecting  ERP  presence,  and  the  ONT  for 
acquiring  high-quality  ERP.  Across  all  experimental  conditions,  the  ANT  mobile  EEG  system 
showed  significantly  higher  levels  of  usability  than  the  other  two  mobile  EEG  systems  when 
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Figure  13.  Comparisons  of  the  MNT  required  for  detecting  ERPs  and  the  ONT  necessary  for 
obtaining  high-quality  ERPs  under  resting  and  active  conditions.  The  horizontal  line  in  the 
center  of  each  notch  indicates  the  median  value,  indicates  p  <  0.0001  in  Wilcoxon  rank 
sum  tests  of  WS  vs.  ANT  and  BP,  and  ABM  vs.  ANT  and  BP  in  both  experimental 
conditions. 
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evaluated  with  both  objective  parameters  (Figure  13).  It  is  thus  evident  that,  at  least  with  respect 
to  the  reliability  of  EEG  signal  measurements  and  the  assurance  of  ERP  quality,  the  ANT  system 
was  superior  to  the  WS  and  ABM  systems.  Furthermore,  the  ANT  system  was  statistically 
comparable  to  the  gold  standard  wired  BP  system. 

V.  Recommendations 

In  the  current  project,  three  mobile  EEG  systems  were  successfully  integrated  into  the  PhyCORE 
at  NHRC  for  testing  and  evaluation.  The  performance  of  these  systems  was  systematically 
evaluated  with  respect  to  their  practical  use  and  capability  for  acquiring  high-quality  EEG 
signals  in  a  highly  dynamic  environment. 

From  a  technical  integration  standpoint,  we  observed  that  the  ABM  system  exhibited  a 
systematic  timing  delay  inherent  to  the  manufacturer’s  system  configuration  (Figure  4,  Tbiue)- 
Although  this  added  delay  could  be  compensated  for  in  off-line  data  analyses,  it  could  be 
potentially  problematic  for  applications  in  which  online  EEG  data  processing  is  required,  such  as 
brain-computer  interface  applications. 

With  regard  to  system  setup,  system  preparation  times  appeared  to  be  a  function  of  both  the 
electrode  type  (i.e.,  dry  vs.  wet)  and  number  of  channels.  The  dry  electrode  WS  system  required 
the  least  amount  of  total  time  to  prepare.  However,  it  should  be  noted  that  the  ANT  system 
(which  features  the  most  channels)  required  the  least  amount  of  time  per  channel  to  prepare.  If 
preparation  time  is  a  consideration,  and/or  not  all  EEG  channels  are  needed  for  a  given  protocol, 
it  is  possible  to  activate  only  those  channels  for  which  the  user  is  most  concerned.  In  such  a 
circumstance,  based  on  the  average  preparation  time  by  channel,  the  ANT  system  offered  the 
most  practical  solution. 

Analyses  of  the  pain  and  comfort  data  indicated  that  the  WS  system  received  higher  pain  ratings 
and  lower  comfort  ratings  than  the  other  two  systems.  Further,  although  the  ABM  and  ANT 
systems  were  rated  equally  low  on  the  pain  scale,  the  ABM  system  received  the  highest  comfort 
ratings  overall. 

With  respect  to  the  reliability  of  EEG  signal  measurement  and  the  assurance  of  ERP  quality,  the 
ANT  system  was  clearly  superior  to  both  the  WS  and  ABM  systems.  Although  all  three  mobile 
systems  demonstrated  the  capability  to  acquire  high-quality  EEG  signals,  significantly  fewer 
experimental  trials  were  needed  when  using  the  ANT  system  to  acquire  the  ERP  signal  in  both 
the  resting  and  active  conditions.  It  is  particularly  notable  that  the  ANT  system  performed  almost 
as  well  as  the  gold  standard  wired  BP  system  in  recording  highly  reliable  EEG  signals  in  the 
resting  condition. 

Based  on  these  findings,  we  recommend  that  of  the  three  mobile  EEG  systems  evaluated,  future 
users  should  pursue  the  ANT  Neuro  eego™sports  64  mobile  EEG  system  as  the  preferred  choice 
for  use  in  complex  and  dynamic  environments.  As  newer  mobile  EEG  systems  are  developed 
and  commercialized,  they  should  be  evaluated  in  a  similar  manner  for  comparison  with  the 
systems  presented  here. 
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